Image display system and image controller

ABSTRACT

An image display system includes a display configured to be attached to the head of a user who is an occupant of a vehicle, to display a subject image to be superimposed on a field of vision of the user; a SLAM camera fixed to the display to capture a SLAM image of surroundings of the display; one or more light on-vehicle displays disposed within a vehicle cabin to emit light that serves as an AR marker; and an image controller configured to determine a display position of the subject image based on the SLAM image including the AR marker.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority to Japanese Patent Application No.2021-189236 filed on Nov. 22, 2021, which is incorporated herein byreference in its entirety including the specification, claims, drawings,and abstract.

TECHNICAL FIELD

The present description discloses an image display system that displaysa subject image to be superimposed on a field of vision of a user or anoccupant of a vehicle and to an image controller.

BACKGROUND

A technique for displaying a predetermined image to be superimposed on auser's field of vision in such a manner as to allow the user to see avirtual object represented by the image as if the virtual objectactually exists has been known. Patent Document 1, for example,discloses a technique of displaying, on smart glasses or a glasses-typedisplay worn by a driver, an image of a leading vehicle that thedriver's vehicle is following. In Patent Document 1, the leading vehicleexpressed by the image moves to guide the driver's vehicle to adestination. This technique enables the driver to travel to thedestination by operating the vehicle so as to follow the leadingvehicle.

CITATION LIST Patent Literature

[Patent Document 1] JP 2017-129406 A

SUMMARY

To enable a user to recognize a virtual object as if the virtual objectwere actually present, it is necessary to determine the position of animage of the virtual object to be displayed on the display (hereinafterreferred to as a “subject image”) based on the position of the virtualobject represented by the subject image in the real space and theposition of the display in the real space.

In Patent Document 1, to identify the position of the display in thereal space, a special marker is disposed on a dashboard, and a camera isattached to the display. The position of the display in the actual spaceis identified based on an image of a scene including the marker that iscaptured by the camera. The use of a marker as in the technique inPatent Document 1 enables detection of the position of the display inthe real space with less computation.

In Patent Document 1, however, the marker is a physical marker, anobject that actually exists. How such a physical marker is viewedsignificantly depends on the surrounding light environmental condition,such as illuminance or color temperatures. This may result in failure todetect the position of the physical marker from the image of thephysical marker captured by the camera on the display. For example, animage of the physical marker captured in a dark environment such as atnighttime may be blacked out, or, in contrast, an image of the physicalmarker captured under strong sunlight may be whited out. In either case,detection of the position of the physical marker may be unsuccessful.Failure to detect the positon of the physical marker inevitably resultsin failure to detect the position of the display in the real space,further resulting in failure to determine the appropriate displayposition of the subject image.

An aspect of the disclosure is therefore aimed toward an image displaysystem that enables more appropriate determination of the displayposition of a subject image, and an image controller.

In accordance with an aspect of the disclosure, an image display systemincludes a display configured to be attached to the head of a user, anoccupant of a vehicle, to display a subject image to be superimposed ona field of vision of the user; a SLAM camera fixed to the display tocapture a SLAM image of surroundings of the display; one or more lightemitters disposed within a vehicle cabin, to emit light that serves as amarker; and an image controller configured to determine a displayposition of the subject image based on the SLAM image including themarker.

A marker created by emitted light can be detected in dark environments,such as at nigh time. This configuration enables reliable identificationof the position of the display in real space and thus enables moreappropriate identification of the display position of the subject image.

In this configuration, the light emitter may be an on-vehicle displaydisposed within the vehicle cabin to display an image, and the markermay be an image displayed in a display area of the on-vehicle display.

This configuration enables changing the shape, position, or brightness,for example, of the marker as desired. This enables providing a markersuitable for the environment within the vehicle to thereby enablefurther reliable identification of the display in real space.

The marker has a marker display condition including at least one of aluminance, a color, or a brightness, and the marker display conditionmay be variable.

Changing the brightness of the marker, for example, increasesdetectability of the marker, which enhances appropriate determination ofthe display position of the subject image.

In the above configuration, the marker display condition may be changedin accordance with environmental light conditions in the vicinity of thelight emitter.

Changing the display condition of the marker in accordance with anenvironmental light condition increases detectability of the marker,which again enhances appropriate determination of the display positionof the subject image.

The image display system may further include a light environment sensorconfigured to detect the environmental light condition in the vicinityof the light emitter, and the image controller may be configured tospecify the environmental light condition in the vicinity of the lightemitter based on a detection result of the light environment sensor.

The light environment sensor enables accurate detection of theenvironmental light condition surrounding the light emitter. This stillfurther enhances appropriate setting of the display condition of themarker.

The image controller may be configured to specify the environmentallight condition in the vicinity of the light emitter based on at leastone of date and time, an illuminating state of a light of the vehicle,or the SLAM image.

This configuration enables detection of the environmental lightcondition surrounding the light emitter without employing the lightenvironment sensor.

The image controller may be configured to change the marker displaycondition in accordance with the SLAM image.

This configuration further increases detectability of the marker.

The image controller may be configured to determine, in response tosuccessful detection of the marker from the SLAM image, the displayposition of the subject image within a display area of the display,based on a virtual position of an object represented by the subjectimage in a real space and a position of the display in the real spacethat is obtained from the SLAM image, and to display the subject imageat the determined position.

This configuration imparts the user with the illusion that the objectrepresented by the subject image actually exists, thus allowingrecognizable augmentation of the user's real environment.

The image controller may be configured to determine, in response tofailure to detect the marker from the SLAM image, the display positionof the subject image within a display area of the display independentlyof a position of the subject image in a real space, and to display thesubject image at the determined position.

This configuration can provide the subject image to the user even afterfailure in detection of the marker.

The image display system may further include an on-vehicle displaydisposed within the vehicle cabin, and the image controller may beconfigured to display, in response to failure to detect the marker fromthe SLAM image, an image corresponding to the subject image on theon-vehicle display.

This configuration can provide an image having information that isequivalent to that of the subject image to the user even in to the eventof a failure to detect the marker.

The marker may be disposed at a position within the field of view of theuser who is sitting on a driver's seat and driving a vehicle.

This configuration enables the user to detect the marker and can providethe subject image while the user is driving.

The one or more light emitters may include two or more light emittersspaced apart from each other, and each of the two or more light emittersmay emit light that serves as the marker.

A plurality of markers increase the accuracy in position detection ofthe display.

In accordance with another aspect of the disclosure, an image controlleris configured to control driving of a display configured to be attachedto the head of a user occupant of a vehicle. The display is configuredto display a subject image to be superimposed on a field of vision ofthe user. The image controller is configured to cause one or more lightemitters disposed within a vehicle cabin to emit light that serves as amarker; cause a SLAM camera fixed to the display to capture a SLAM imageof surroundings of the display; and determine a display position of thesubject image based on the SLAM image including the marker.

The marker created by emitted light can be detected in darkenvironments, such as at nighttime. This configuration enables reliableidentification of the position of the display in real space, and thusenables more appropriate identification of the display position of thesubject image.

The technique of the disclosure enables more appropriate determinationof the display position of the subject image.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments of the present disclosure will be described based on thefollowing figures, wherein:

FIG. 1 is a block diagram illustrating a configuration of an imagedisplay system;

FIG. 2 illustrates a user wearing a wearable device;

FIG. 3 schematically illustrates a field of view of a driver or a user;

FIG. 4 is an image view for explaining a space-fixed display mode anddevice-fixed display mode;

FIG. 5 schematically illustrates a field of view of a user with asubject image being displayed;

FIG. 6 schematically illustrates a field of view of a driver duringnight time;

FIG. 7 illustrates another example AR marker;

FIG. 8 illustrates an example brightness profile;

FIG. 9 illustrates example image display in response to failure todetect an AR marker;

FIG. 10 is a flowchart illustrating a flow of image display processingin the image display system; and

FIG. 11 is a flowchart illustrating a flow of visual SLAM processing.

DESCRIPTION OF EMBODIMENTS

The configuration of an image display system 10 will be described byreference to the drawings. While in the following specific embodimentsare described for the ease of understanding, these are only examples andmay be modified as appropriate. FIG. 1 is a block diagram illustratingthe configuration of the image display system 10. The image displaysystem 10 includes a wearable device 12 and an on-vehicle system 28.

The wearable device 12 is a device an occupant of the vehicle, such as adriver, wears on their head and may be an eyeglass or goggle shapeddevice. The wearable device 12 includes a display 14, a SLAM camera 16,a pupil position sensor 18, and a device controller 20.

The wearable device 12 will be described in detail with reference toFIG. 2 . FIG. 2 illustrates a user 100 occupant of the vehicle wearingthe wearable device 12. The wearable device 12 is in the form ofeyeglasses and is referred to as smart glasses or AR glasses. Thewearable device 12 includes temples 26 that are linear frames to be putover respective ears and a rim 24 that is a frame surrounding the eyesand that is to be put across the nose.

The display 14 displays an image on the field of vision of the user 100wearing the wearable device 12. In this example, the display 14 is anorganic EL display or a liquid crystal display having a display area 22disposed within the rim 24, and displays an image in part of or over theentire region of the display area 22. The display area 22 having hightransparency allows the user 100 or the occupant to visually recognizescenery in front through the display area 22 with no image beingdisplayed on the display area 22. The display area 22, when displayingan image in only part of the display area 22, allows the user 100 to seeboth the forward scenery in the field of view and the displayed imagesimultaneously. This image may at this time be opaque or translucent. Inthe following description, an image displayed on the display 14 isreferred to as a “subject image” for discrimination from other images.Also, an object which is virtually represented by the subject image isreferred to as a “subject”.

The SLAM camera 16 is fixed to the display 14 to image the surroundingsof the display 14. The SLAM camera 16 is secured, for example, to thevicinity of a front end of the temple 26 so as to face forward, andcaptures an image within a range similar to that of the field of visionof the user 100. In the following description, an image captured by theSLAM camera 16 will be referred to as a “SLAM image”. An imagecontroller 30, which will be described below, specifies the position andattitude of the display 14 in real space based on an AR marker in theSLAM image, as will be described below.

The pupil position sensor 18 detects the positions of pupils in theright and left eyes of the user 100, and is fixed to the vicinity of thecenter of the rim 24, for example. The pupil position sensor 18 may beformed of a camera, for example.

The device controller 20 controls operation of the wearable device 12 inresponse to an instruction from the image controller 30. The devicecontroller 20 may be a computer having a processor and a memory, forexample. The device controller 20 continuously transmits the imagescaptured by the SLAM camera 16 and the pupil position sensor 18 to theimage controller 30 and displays the subject image on the display 14 inaccordance with an instruction from the image controller 30.

Referring again to FIG. 1 , the on-vehicle system 28 will be described.The on-vehicle system 28 is installed in a vehicle, and includes theimage controller 30, a meter display 40 a, a multi display 40 b, anelectronic inner mirror 40 c, and a light environment sensor 42. Themeter display 40 a, the multi display 40 b, and the electronic innermirror 40 c are installed in a vehicle and can be visually recognized bythe driver during driving. In the following description, these displayswill be referred to as “on-vehicle displays 40” unless discriminationamong these displays is necessary. The on-vehicle displays 40 functionas light emitters that emit light forming the AR marker, as will bedescribed below.

The arrangement of the on-vehicle displays 40 will be described byreference to FIG. 3 . FIG. 3 schematically illustrates the field of viewof the driver or the user 100. The meter display 40 a indicatesinformation regarding the state of the vehicle, such as the speed andmileage. As illustrated in FIG. 3 , the meter display 40 a is disposedon the opposite side of the steering wheel 56 from the driver, whichenables the driver to visually recognize the display area of the meterdisplay 40 a through the steering wheel 56.

The multi display 40 b indicates information regarding on-vehicleelectronic instruments, such as a navigation device or an audio device.As illustrated in FIG. 3 , the multi display 40 b is disposed at thecenter of the instrument panel in the vehicle width direction, that is,on what is commonly called a center console.

The electronic inner mirror 40 c displays an image from the rear of thevehicle as imaged by a rearview camera (not shown). The electronic innermirror 40 c is used in place of a rearview mirror which shows the rearof the vehicle by optical reflection. The electronic inner mirror 40 cmay be switchable between a digital mode showing an image and a mirrormode for showing the rear by optical reflection. As illustrated in FIG.3 , the electronic inner mirror 40 c is disposed at a position where arearview mirror is typically disposed, that is, in the vicinity of theupper end of the windshield glass.

Referring again to FIG. 1 , the light environment sensor 42 detects anenvironmental light condition around the on-vehicle displays 40. Anenvironmental light condition refers to conditions including at leastone of luminance or color temperature of light. The light environmentsensor 42 may include at least one of an illuminance sensor that detectslightness of light or a color temperature sensor that detects color oflight. A single light environment sensor 42 or two or more lightenvironment sensors 42 may be disposed. For example, the lightenvironment sensor 42 may be disposed on the wearable device 12. Inanother embodiment, the light environment sensors 42 may be disposednear the respective on-vehicle displays 40. The light environment sensor42 may be disposed specifically for the image display system 10, or anexisting sensor installed in a vehicle may be used for the lightenvironment sensor 42. For example, some vehicles include anauto-lighting function to automatically turn on lights such as aheadlight in response to darkness around the vehicle, and an illuminancesensor that is an auto-lighting sensor disposed for such anauto-lighting function may be used as the light environment sensor 42.Further, some multi displays 40 b contain an illuminance sensor toautomatically adjust the emission brightness in accordance with theperipheral lightness, and such an illuminance sensor included in themulti displays 40 b may be used as the light environment sensor 42.

The image controller 30 generates data of a subject image to bedisplayed on the display 14. The image controller 30 is physically acomputer including a processor 32, a memory 34, and a communication I/F35. The computer includes a microcontroller composed of a computersystem integrated into a single integrated circuit. The processor 32refers to a processor in a broad sense, and includes a general-purposeprocessor, such as a Central Processing Unit (CPU), and aspecial-purpose processor, such as a Graphics Processing Unit (GPU), anApplication Specific Integrated Circuit (ASIC), a Field ProgrammableGate Array (FPGA), or a programmable logic device.

The memory 34 refers to a device that stores digital data to beprocessed by a computer. The memory 34 includes at least one of a mainmemory connected with the processor 32 via a memory bus and a secondarymemory device that accesses the processor 32 via an input/outputchannel. The memory 34 may include at least one of a semiconductormemory, such as RAM, ROM, or solid state drive, for example, or amagnetic disk, such as a hard disk drive.

The communication I/F 35 transmits and receives data, through wire orwirelessly, to and from other electronic devices or specifically thewearable device 12, the on-vehicle displays 40, and the lightenvironment sensor 42. For example, the communication I/F 35 maytransmit and receive data to and from the on-vehicle displays 40 and thelight environment sensor 42 through Controller Area Network (CAN)communication. The communication I/F 35 may further transmit and receivedata to and from the wearable device 12 through short-distance wirelesscommunication such as Bluetooth (registered mark), Wi-Fi (registeredmark), or infrared communication.

The image controller 30 may include a plurality of computersmechanically spaced from each other, rather than being a singlecomputer. The processing of the image controller 30 which will bedescribed below may be partially implemented by a computer installed inthe wearable device 12 or the on-vehicle displays 40. While in thepresent embodiment, the image controller 30 is mounted in the vehicle,the image controller 30 may be incorporated in the wearable device 12 orin a mobile computer, such as a smartphone, which is separate from thewearable device 12.

The image controller 30 generates data of the subject image to bedisplayed on the display 14, as described above. The display 14 displaysthe subject image either in a “space-fixed display” mode or a“device-fixed display” mode. These two display modes will be describedwith reference to FIG. 4 .

In the space-fixed display mode, a subject image that represents apredetermined object is displayed as if the object exists in real space.In the device-fixed display mode, a subject image that represents apredetermined object is displayed at a specific position in the displayarea 22 irrespective of the position of the object in real space.

Assuming, for example, that the user 100 is viewing a real space thatactually includes a table 80 through the display area 22 of the display14, as illustrated in FIG. 4 . In this case, displaying a subject image50 that represents a “ball” in the display area 22 would naturallyresult in the field of vision of the user 100 simultaneously showing thereal space including the table 80 and the subject image 50 of the“ball”, as indicated in a state S1 in FIG. 4 .

In the device-fixed display mode, the display position of a subject 72indicated by the subject image 50, which is a “ball” in the example ofFIG. 4 , is determined independently of its position in the real space.Therefore, in the device-fixed display mode, moving the eyepoint of theuser 100 would not require changes in the display position, size, andshape of the subject image 50 within the display area 22 as indicted ina state S2 in FIG. 4 .

In the space-fixed display mode, the place in the real space where thesubject 72 indicated by the subject image 50, which is a “ball” in theexample of FIG. 4 , is located is specified, and the subject image 50 isdisplayed as if the subject 72 is actually present at the specifiedposition. For example, assuming that, in the space-fixed display mode,the subject 72 or a “ball” is located on the table 80 in the real space.In this case, the display position, size, and shape of the “ball” withinthe display area 22 are changed such that the user 100 can view the“ball” on the table 80 even if the eyepoint of the user 100 moves asindicated in a state S3.

As described above, displaying the subject image 50 in the space-fixeddisplay mode gives the user 100 the illusion that the subject 72represented by the subject image 50 is actually present. In other words,displaying the subject image 50 in the space-fixed display mode wouldenable addition, deletion, emphasis, and attenuation of information withrespect to the real-world environment, thereby augmenting the real worldviewed by humans. This technique is typically referred to as “augmentedreality” or “AR”.

An example display of the subject image 50 in the present embodimentwill be now described. FIG. 5 schematically illustrates the field ofvision of the user 100 (or a driver in this example) with subject images50 a, 50 b, and 50 c being displayed. In the example illustrated in FIG.5 , the subject image 50 a for emphasizing a pedestrian, the subjectimage 50 b indicating the travelling direction of the vehicle, and thesubject image 50 c indicating message that attracts driver's attention,are displayed in the space-fixed display mode. These subject images 50a, 50 b, and 50 c are displayed on the display 14 at positions and insizes that are similar to the positions and sizes of the subjectsrepresented by the respective images existing in the real world. Thesubject image 50 a, for example, is displayed in the display area 22 atthe position and in the size of an object moving with the pedestrianassuming that the object actually exists in the real world. Therefore,the position and the size of the subject image 50 a within the displayarea 22 changes with the change of the relative positional relationshipbetween the pedestrian and the user 100. The subject image 50 a may alsochange its shape in accordance with the position or posture of thepedestrian in the real world.

The subject image 50 b is displayed in the display area 22 at theposition and in the size similar to those of an arrow-shape objectrepresented by the subject image 50 b assuming that the object actuallyresides on the road surface in front of the vehicle that actuallyexists. The subject image 50 c is displayed in the display area 22 at aposition and in a size similar to those of a text box represented by thesubject image 50 c assuming that the object actually resides on theupper right portion of the steering wheel 56 in the real world. Thus,the display positions and the sizes of the subject images 50 b and 50 cwithin the display area 22 change with the movement of the eyepoint ofthe user 100.

As described above, the space-fixed display mode enables display of thesubject image 50 in consideration of locations of actual objects,thereby reliably preventing the subject image 50 from obstructing thedriving operation. The space-fixed display mode further enables thesubject image 50 to be displayed at a position correlated with that ofthe actual object, such as a pedestrian, thus effectively directingattention of the user 100 toward the object.

To achieve the space-fixed display, it is necessary to accurately detectthe position of the pupils with respect to the display 14, and theposition and attitude of the display 14 in the real space. Based on theposition and attitude of the subject in the real space, the position andattitude of the display 14 in the real space, and the positions of thepupils in the display 14, the image controller 30 determines theposition of, for example, the subject image 50 within the display area22. As described above, the positions of the pupils with respect to thedisplay 14 are detected by the pupil position sensor 18.

The position and attitude of the display 14 in the real space iscalculated by the image controller 30 performing Visual SimultaneousLocalization and Mapping (visual SLAM) based on a SLAM image captured bythe SLAM camera 16. Visual SLAM refers to a technique for estimatingthree-dimensional information of the environment and the position andattitude of the camera simultaneously based on images captured by thecamera. To facilitate performing of visual SLAM, an AR marker 60 (seeFIG. 3 ) is disposed within the vehicle. To calculate the position andattitude of the display 14 in the real space based on the coordinates,size, and distortion, for example, of the image of the AR marker 60within the SLAM image, the image controller 30 extracts an image of theAR marker 60 from the SLAM image captured by the SLAM camera 16.

While an actual object has been used for such an AR marker 60, in somecases, the image controller 30 is unable to detect an actual objectbecause of the environmental light conditions within the vehicle. Forexample, in a dark environment such as during nighttime, an area aroundthe AR marker 60 may be clipped in the SLAM image and may appear as ablack area. In contrast, the AR marker 60 in strong sunlight may appearas a white area when shown in the SLAM image, which is referred to asblown-out highlights. In the case of such clipping, the image controller30 is unable to detect the AR marker 60 or the position and attitude ofthe display 14 in the real space, failing to perform space-fixed displayof the subject image 50.

In this embodiment, the AR marker 60 for use in visual SLAM is composedof light emitted from the on-vehicle displays 40 (or light emitters).Specifically, in this embodiment, the on-vehicle displays 40 are causedto display an image that functions as the AR marker 60. In the exampleillustrated in FIG. 3 , cross-shape images displayed on the meterdisplay 40 a, the multi display 40 b, and the electronic inner mirror 40c, respectively, function as the AR marker 60 for use in visual SLAM.

The AR markers 60 formed of light emitted from the on-vehicle displays40 as described above appropriately appear in the SLAM image in a darkenvironment such as at nighttime, as illustrated in FIG. 6 . Thisenables the image controller 30 to perform visual SLAM properly based onthe SLAM image, and to thereby display the subject image 50appropriately.

The image controller 30 is able to obtain the shape and display positionof the AR marker 60, which may be fixed or change as appropriate, at arequired timing. For example, the display position and shape of the ARmarker 60 may be predetermined and fixed. In this configuration, theimage controller 30 prestores the determined display position and shapeof the AR marker 60. In another embodiment, the display position andshape of the AR marker 60 may be changed as appropriate. For example,the AR marker 60 may be displayed on the upper right corner of the multidisplay 40 b which is showing map information and may be displayed onthe lower right corner of the multi display 40 b which is showing audioinformation. Further, the shape of the AR marker 60 may be changedbetween these two cases. In this configuration, one of the imagecontroller 30 or the on-vehicle display 40 determines the displayposition and shape of the AR marker 60, and transmits the determinedinformation to the other through data communication.

The AR markers 60 displayed on the different on-vehicle displays 40 mayhave an identical shape or different shapes. Specifically, asillustrated in FIG. 7 , the three on-vehicle displays 40 a, 40 b, and 40c may respectively display the AR markers 60 a, 60 b, and 60 c havingdifferent shapes. The AR marker 60 may be provided specifically for usein visual SLAM, or an existing image may be used for the AR marker 60.For example, the meter display 40 a displays an image representing thespeed unit, which is Km/h, in the example illustrated in FIG. 7 ,irrespective of execution of visual SLAM, and this image may be used asthe AR marker 60 a. Not just visible light but also invisible light,which can be detected by the SLAM camera 16, may form the AR marker 60.For example, an image of the AR marker 60 may be formed based oninfrared light.

The image controller 30 may determine the marker display conditionincluding at least one of luminance, color, or brightness in accordancewith the environmental light conditions in the vicinity of theon-vehicle display 40 and instruct the on-vehicle display 40 to displaythe AR marker 60 under the determined marker display condition. Forexample, the image controller 30 may change the display luminance of theAR marker 60 in accordance with the illuminance in the vicinity of theon-vehicle display 40 (hereinafter referred to as “environmentalilluminance”). In this configuration, the image controller 30 mayprestore the luminance profile as illustrated in FIG. 8 and determinethe display luminance of the AR marker 60 based on this luminanceprofile. In the luminance profile illustrated in FIG. 8 , the horizontalaxis indicates the environmental illuminance and the vertical axisindicates the display luminance of the AR marker 60. In the exampleillustrated in FIG. 8 , the higher the environmental illuminance, thehigher the display luminance of the AR marker 60. This configurationallows the AR marker 60 to be clearly displayed under the environmentwith high environmental illuminance. Meanwhile, in the environment withlow environmental illuminance, it is possible to prevent the AR marker60 from being excessively bright, avoiding the user 100 from beingdazzled.

In another embodiment, the image controller 30 may change at least oneof color or brightness of the AR marker 60 in accordance with the colortemperature in the vicinity of the on-vehicle display 40 (hereinafterreferred to as “environmental color temperature”). For example, when theenvironmental color temperature is a color temperature with sunset-likestrong red, the AR marker 60 may be changed to a color closer to blue.

The environmental light conditions in the vicinity of the on-vehicledisplay 40 may also be specified based on the detection result of thelight environment sensor 42. In another embodiment, the image controller30 may estimate the environmental light conditions in the vicinity ofthe on-vehicle display 40 based on the date and time or the illuminatingstate of vehicle lights. For example, the image controller 30 maycalculate the solar altitude based on the date and time and estimate theenvironmental light conditions in the vicinity of the on-vehicle display40, such as illuminance, based on the altitude. The image controller 30may further modify the environmental light conditions estimated from thedate and time, based on at least one of the weather, the vehicleposition, or the vehicle orientation. For example, the image controller30 may estimate the intensity of the sunlight based on the weather andmodify the environmental light conditions estimated from the date andtime, based on the estimation result. The image controller 30 mayfurther estimate the degree of sunlight shielding, such as whether thevehicle is located indoors, based on the vehicle position, and modifythe environmental light conditions estimated from the date and timebased on the estimation result. The image controller 30 may alsoestimate whether the vehicle cabin is illuminated by direct sunlightbased on the solar altitude and the vehicle orientation, and modify theenvironmental light conditions estimated from the date and time based onthe estimation result. In another embodiment, the image controller 30may estimate the environmental light conditions in the vicinity of theon-vehicle display 40 based on the illuminating state of lights that areobliged to be turned on in the nighttime, such as headlights.

In another embodiment, the image controller 30 may estimate theenvironmental light conditions in the vicinity of the on-vehicle display40 based on the SLAM image captured by the SLAM camera 16. As thewearable device 12 is attached to the head of the vehicle occupant, theSLAM image can be assumed to be an image of the vehicle interior. It ishighly likely that the luminance and color of the entire SLAM imagereflects the environmental light conditions of the vehicle interior, andthus, in the vicinity of the on-vehicle display 40. The image controller30 may therefore estimate the environmental light conditions in thevicinity of the on-vehicle display 40 based on the trend of luminanceand color of the entire SLAM image.

While in the above description, the display condition of the AR marker60 is changed based on the environmental light conditions in thevicinity of the on-vehicle display 40, the image controller 30 maychange the display condition of the AR marker 60 based on the SLAM imagecaptured by the SLAM camera 16. For example, it is highly likely thatfailure to extract the AR marker 60 from the SLAM image occurs fromshortage of display luminance of the AR marker 60. Therefore, when it isnot possible to extract the AR marker 60 from the SLAM image, the imagecontroller 30 may instruct the on-vehicle display 40 to increase thedisplay luminance of the AR marker 60. Further, in response to failureto extract the AR marker 60 from the SLAM image, the image controller 30may instruct the on-vehicle display 40 to gradually change the displayluminance and the color of the AR marker 60 in one direction and specifythe display luminance, for example, upon proper detection of the ARmarker 60, as the display condition of the AR marker 60. When the ARmarker 60 extracted from the SLAM image include an edge that is notsufficiently clear, the image controller 30 may instruct the on-vehicledisplay 40 to change the display condition of the AR marker 60 so as todisplay the edge clearly.

As described above, in the present embodiment, the image displayed onthe on-vehicle display 40 is used as the AR marker 60. Thisconfiguration enables the image controller 30 to detect the AR marker 60more reliably to thereby execute visual SLAM more properly and thusdisplay the subject image 50 more properly. Under certain environmentallight conditions within the vehicle, the image controller 30 fails todetect the AR marker 60 even after the luminance, for example, of the ARmarker 60 has been changed. In this case, the image controller 30 maydisplay the subject image 50 in the device-fixed display mode, ratherthan the space-fixed display mode. In another embodiment, in response tofailure to detect the AR marker 60, the image controller 30 may displayan image corresponding to the subject image 50 on the on-vehicle display40.

Assuming, for example, a case wherein it is attempted to display asubject image 50 a to direct the attention to a pedestrian, but the ARmarker 60 cannot be detected, the image controller 30 may display, onthe multi display 40 b, an image that promotes attention to thepedestrian, as illustrated in FIG. 9 . The image controller 30 mayfurther display, on a specific location in the display area 22, an imagethat promotes attention to the pedestrian (the subject image 50 a inFIG. 9 ). In displaying the subject image 50 a in the device-fixeddisplay mode, consideration should be made to prevent the image 50 afrom obstructing the driving operation. For example, as humans tend toacquire various information necessary for driving operation from thecenter of their field of vision, there is significant risk that an imagedisplayed in the center of the field of vision may obstruct the drivingoperation. Therefore, in displaying the subject image 50 a in thedevice-fixed display mode, the image 50 a may be displayed at a cornerof the display area 22. Further, in displaying the subject image 50 a inthe device-fixed display mode, the image 50 a may be translucent so asto allow the driver to visually recognize the surroundings through theimage 50 a.

Referring now to FIG. 10 , the flow of image displaying processing inthe image display system 10 will be described. In response to adetermination that display of the subject image 50 is necessary (Yes instep S10), the image controller 30 performs visual SLAM processing tospecify the position and attitude of the display 14 in the real space(S12). The visual SLAM processing will be described in detail below.

In response to success of the visual SLAM processing (Yes in step S14),the image controller 30, based on the specified position and attitude ofthe display 14 in the real space, specifies the display position, forexample, of the subject image 50 in the display area 22 (step S16) anddisplays the subject image 50 on the display 14 in the space-fixeddisplay mode (step S18). Thereafter, the process returns to step S10,and similar processing is repeated.

In response to failure of the visual SLAM processing (No in step S14),the image controller 30 displays an image corresponding to the subjectimage 50 on the display 14 in the device-fixed display mode or displaysthe image on the on-vehicle display 40 (S20). Thereafter, the processreturns to step S10 to repeat similar processing.

Referring now to FIG. 11 , the flow of the visual SLAM processing willbe described. To perform the visual SLAM processing, the imagecontroller 30 specifies the environmental light conditions in thevicinity of the on-vehicle display 40 (S30). The environmental lightingcondition may be specified based on the detection result of the lightenvironment sensor 42 or based on the date and time, the illuminatingstate of lights, and the SLAM image, for example.

After the environmental light condition is specified, the imagecontroller 30, based on the environmental light conditions, determinesthe display condition, such as display luminance or color, of the ARmarker 60 (step S32), and instructs the on-vehicle display 40 to displaythe AR marker 60 under the determined display condition (step S34). Inresponse to this instruction, the on-vehicle display 40 displays the ARmarker 60.

Subsequently, the image controller 30 acquires the SLAM image capturedby the SLAM camera 16 (step S36). The image controller 30 furtherdetermines whether the AR marker 60 can be detected from the SLAM image(step S38). In response to the determination that the AR marker 60 canbe detected (Yes in step S38), the image controller 30 specifies theposition, size, and distortion, for example, of the AR marker 60 withinthe SLAM image (step S40), and further calculates, based on thespecified information of the AR marker 60, the position and attitude ofthe display 14 in the real space (step S42).

In response to the determination that the AR marker 60 cannot bedetected from the SLAM image in step S38 (No in step S38), the imagecontroller 30 proceeds to step S20 (see FIG. 10 ) without calculatingthe position or attitude of the display 14.

As is clear from the above description, in the present embodiment, theAR marker 60 that is necessary for visual SLAM is formed of lightemitted from the light emitter, which is specifically the on-vehicledisplay 40. This configuration enables detection of the AR marker 60 andthus appropriate performing of visual SLAM under the dark environmentsuch as in the nighttime. Further, in the present embodiment, to enableproper detection of the AR marker 60, the display condition of the ARmarker 60 is changed based on the environmental light conditions in thevicinity of the on-vehicle display 40 or the SLAM image that iscaptured. This configuration enables more reliable detection of the ARmarker 60 and thus proper performing of visual SLAM with the change ofthe environmental light conditions in the vicinity of the on-vehicledisplay 40.

The above description describes only examples, and, while the imagedisplay system should at a minimum be configured to use an image formedof light emitted from a light emitter as the AR marker 60, theconfiguration of other elements may be modified. For example, in theabove description, the on-vehicle display 40 that displays an image isused as the light emitter, other devices, such as an indicator lamp andillumination devices disposed within the vehicle cabin to emit light ofa predetermined mode, may be used as the light emitter.

Further, while in the above description, the display 14 shows an imageon the display area 22, the display 14 may be a projector that projectsan image onto the retina of the user 100. Further, while in the abovedescription, the user 100 visually recognizes the real space through thetransparent display area 22, the display area 22 may be configured to beopaque to prevent the user 100 from visually recognizing the real spacethrough the display area 22. In this configuration, the image controller30 displays, on the display area 22, a synthesis image including animage of the real space and a subject image representing a virtualobject.

REFERENCE SIGNS LIST

10 image display system, 12 wearable device, 14 display, 16 SLAM camera,18 pupil position sensor, 20 device controller, 22 display area, 24 rim,26 temple, 28 on-vehicle system, 30 image controller, 32 processor, 34memory, 35 communication I/F, 40 on-vehicle display, 40 a meter display,40 b multi display, 40 c electronic inner mirror, 42 light environmentsensor, 50 subject image, 56 steering wheel, 60 AR marker, 72 subject,80 table, 100 user.

1. An image display system, comprising: a display configured to beattached to the head of a user that is an occupant of a vehicle, thedisplay configured to display a subject image to be superimposed on afield of vision of the user; a SLAM camera fixed to the display tocapture a SLAM image of surroundings of the display; one or more lightemitters disposed within a vehicle cabin, the one or more light emittersconfigured to emit light that serves as a marker; and an imagecontroller configured to determine a display position of the subjectimage based on the SLAM image including the marker.
 2. The image displaysystem according to claim 1, wherein the light emitter is an on-vehicledisplay disposed within the vehicle cabin to display an image, and themarker is an image displayed in a display area of the on-vehicledisplay.
 3. The image display system according to claim 1, wherein themarker has a marker display condition including at least one of aluminance, a color, or a brightness, the marker display condition beingvariable.
 4. The image display system according to claim 3, wherein themarker display condition is changed in accordance with an environmentallight condition in the vicinity of the light emitter.
 5. The imagedisplay system according to claim 4, further comprising: a lightenvironment sensor configured to detect the environmental lightcondition in the vicinity of the light emitter, wherein the imagecontroller is configured to specify the environmental light condition inthe vicinity of the light emitter based on a detection result of thelight environment sensor.
 6. The image display system according to claim4, wherein the image controller is configured to specify theenvironmental light condition in the vicinity of the light emitter basedon at least one of a date and time, an illuminating state of a light ofthe vehicle, or the SLAM image.
 7. The image display system according toclaim 3, wherein the image controller is configured to change the markerdisplay condition in accordance with the SLAM image.
 8. The imagedisplay system according to claim 1, wherein in response to success indetection of the marker from the SLAM image, the image controller isconfigured to determine the display position of the subject image withina display area of the display, based on a virtual position of an objectrepresented by the subject image in a real space and a position of thedisplay in the real space that is obtained from the SLAM image, and todisplay the subject image at the determined position.
 9. The imagedisplay system according to claim 1, wherein in response to failure todetect the marker from the SLAM image, the image controller isconfigured to determine the display position of the subject image withina display area of the display independently of a position of the subjectimage in a real space, and to display the subject image at thedetermined position.
 10. The image display system according to claim 1,further comprising: an on-vehicle display disposed within the vehiclecabin, wherein in response to failure to detect the marker from the SLAMimage, the image controller is configured to display an imagecorresponding to the subject image on the on-vehicle display.
 11. Theimage display system according to claim 1, wherein: the marker isdisposed at a position within the field of view of the user who issitting on a driver's seat and driving a vehicle.
 12. The image displaysystem according to claim 1, wherein the one or more light emitterscomprise two or more light emitters spaced from each other, and each ofthe two or more light emitters emits light that serves as the marker.13. An image controller configured to control driving of a displayconfigured to be attached to the head of a user that is an occupant of avehicle, the display configured to display a subject image to besuperimposed on a field of vision of the user, the image controllerconfigured to: cause one or more light emitters disposed within avehicle cabin to emit light that serves as a marker; cause a SLAM camerafixed to the display to capture a SLAM image of surroundings of thedisplay; and determine a display position of the subject image based onthe SLAM image including the marker.